Display of Computer Generated Image of an Out-of-View Portion of a Medical Device Adjacent a Real-Time Image of an In-View Portion of the Medical Device

ABSTRACT

Systems and methods for performing robotically-assisted surgical procedures on a patient enable an image display device to provide an operator with auxiliary information related to the surgical procedure, in addition to providing an image of the surgical site itself. The systems and methods allow an operator to selectively access and reference auxiliary information on the image display device during the performance of a surgical procedure.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. application Ser. No. 12/943,754filed Nov. 10, 2010, which is a divisional of U.S. application Ser. No.11/093,372 filed Mar. 30, 2005, each of which is incorporated herein byreference.

U.S. application Ser. No. 11/093,372 is a continuation-in-part of U.S.application Ser. No. 10/314,001 filed Dec. 5, 2002, now U.S. Pat. No.7,107,090, which is a continuation of U.S. application Ser. No.09/464,455 filed Dec. 14, 1999, now U.S. Pat. No. 6,522,906, which is acontinuation-in-part of U.S. application Ser. No. 09/457,406, filed Dec.7, 1999, now U.S. Pat. No. 6,799,065, each of which is incorporatedherein by reference.

U.S. application Ser. No. 11/093,372 is additionally acontinuation-in-part of U.S. application Ser. No. 10/644,406, filed Aug.19, 2003, now abandoned, which is a continuation of U.S. applicationSer. No. 10/163,626, filed Jun. 5, 2002, now U.S. Pat. No. 6,671,581,which is a continuation of U.S. application Ser. No. 09/373,678, filedAug. 13, 1999, now U.S. Pat. No. 6,424,885, which claims benefit of U.S.Provisional Application No. 60/128,160, filed Apr. 7, 1999, each ofwhich is incorporated herein by reference.

BACKGROUND OF THE INVENTION

The present invention is generally related to improved robotic devices,systems and methods, for use in telerobotic surgery.

Minimally invasive medical techniques are aimed at reducing the amountof extraneous tissue which may be damaged during diagnostic or surgicalprocedures, thereby reducing patient recovery time, discomfort, anddeleterious side effects. Many surgeries are performed each year in theUnited States. A significant amount of these surgeries can potentiallybe performed in a minimally invasive manner. However, only a relativelysmall percentage of surgeries currently use minimally invasivetechniques due to limitations of minimally invasive surgical instrumentsand techniques currently used and the difficulty experienced inperforming surgeries using such traditional instruments and techniques.

Advances in minimally invasive surgical technology could dramaticallyincrease the number of surgeries performed in a minimally invasivemanner. The average length of a hospital stay for a standard surgery issignificantly longer than the average length for the equivalent surgeryperformed in a minimally invasive surgical manner. Thus, expansion inthe use of minimally invasive techniques could save millions of hospitaldays, and consequently millions of dollars annually, in hospitalresidency costs alone. Patient recovery times, patient discomfort,surgical side effects, and time away from work can also be reduced byexpanding the use of minimally invasive surgery.

Traditional forms of minimally invasive surgery include endoscopy. Oneof the more common forms of endoscopy is laparoscopy, which is minimallyinvasive inspection or surgery within the abdominal cavity. Intraditional laparoscopic surgery a patient's abdominal cavity isinsufflated with gas and cannula sleeves are passed through small(approximately ½ inch) incisions in the musculature of the patient'sabdomen to provide entry ports through which laparoscopic surgicalinstruments can be passed in a sealed fashion.

The laparoscopic surgical instruments generally include a laparoscopefor viewing the surgical field and working tools defining end effectors.Typical surgical end effectors include clamps, graspers, scissors,staplers, and needle holders, for example. The working tools are similarto those used in conventional (open) surgery, except that the workingend or end effector of each tool is separated from its handle by anapproximately 12-inch long extension tube, for example, so as to permitthe surgeon to introduce the end effector to the surgical site and tocontrol movement of the end effector relative to the surgical site fromoutside a patient's body.

To perform surgical procedures, the surgeon typically passes theseworking tools or instruments through the cannula sleeves to the internalsurgical site and manipulates the instruments or tools from outside theabdomen by sliding them in and out through the cannula sleeves, rotatingthem in the cannula sleeves, levering (i.e., pivoting) the instrumentsagainst the abdominal wall and actuating the end effectors on the distalends of the instruments from outside the abdominal cavity. Theinstruments normally pivot around centers defined by the incisions whichextend through the muscles of the abdominal wall. The surgeon typicallymonitors the procedure by means of a television monitor which displaysan image of the surgical site via the laparoscopic camera. Typically,the laparoscopic camera is also introduced through the abdominal wall soas to capture an image of the surgical site. Similar endoscopictechniques are employed in, e.g., arthroscopy, retropentoneoscopy,pelviscopy, nephroscopy, cystoscopy, cistemoscopy, sinoscopy,hysteroscopy, urethroscopy, and the like.

There are many disadvantages relating to such traditional minimallyinvasive surgical (MIS) techniques. For example, existing MISinstruments deny the surgeon the flexibility of tool placement found inopen surgery. Difficulty is experienced in approaching the surgical sitewith the instruments through the small incisions. The length andconstruction of many endoscopic instruments reduces the surgeon'sability to feel forces exerted by tissues and organs on the end effectorof the associated instrument. Furthermore, coordination of the movementof the end effector of the instrument as viewed in the image on thetelevision monitor with actual end effector movement is particularlydifficult, since the movement as perceived in the image normally doesnot correspond intuitively with the actual end effector movement.Accordingly, lack of intuitive response to surgical instrument movementinput is often experienced. Such a lack of intuitiveness, dexterity andsensitivity of endoscopic tools has been found to be an impediment tothe expansion of the use of minimally invasive surgery.

Minimally invasive telesurgical systems for use in surgery have been andare still being developed to increase a surgeon's dexterity as well asto permit a surgeon to operate on a patient in an intuitive manner.Telesurgery is a general term for surgical systems where the surgeonuses some form of remote control, e.g., a servomechanism, or the like,to manipulate surgical instrument movements, rather than directlyholding and moving the tools by hand. In such a telesurgery system, thesurgeon is typically provided with an image of the surgical site on avisual display at a location remote from the patient. The surgeon cantypically perform the surgical procedure at the location remote from thepatient whilst viewing the end effector movement during the surgicalprocedure on the visual display. While viewing typically athree-dimensional image of the surgical site on the visual display, thesurgeon performs the surgical procedures on the patient by manipulatingmaster control devices at the remote location, which master controldevices control motion of the remotely controlled instruments.

Typically, such a telesurgery system can be provided with at least twomaster control devices (one for each of the surgeon's hands), which arenormally operatively associated with two robotic arms on each of which asurgical instrument is mounted. Operative communication between mastercontrol devices and associated robotic arm and instrument assemblies istypically achieved through a control system. The control systemtypically includes at least one processor which relays input commandsfrom the master control devices to the associated robotic arm andinstrument assemblies and from the arm and instrument assemblies to theassociated master control devices in the case of, e.g., force feedback,or the like.

One object of the present invention is to provide improved telesurgerysystems, devices and methods for use in surgery. Another object of theinvention is to provide a telesurgical system and method wherebyauxiliary information related to a surgical procedure to be performed bythe telesurgical system can be selectively displayed on a viewer of thesystem, together with an image of the surgical site captured by an imagecapture device, such as an endoscope, of the system, so as to enable anoperator of the system selectively to reference such auxiliaryinformation on the viewer during the performance of the surgicalprocedure. In this manner the surgical procedure can typically beperformed with greater confidence, safety, efficacy and in some casesgreater accuracy.

SUMMARY OF THE INVENTION

According to one aspect of the invention, there is provided a medicalsystem comprising: an endoscope capturing real-time images of a surgicalsite; a viewer displaying the real-time images; a surgical instrumenthaving an end effector; and a processor programmed to cause a computergenerated image of the end effector to be displayed on the viewer in anauxiliary image surrounding the displayed real-time images so as toindicate a position of the end effector relative to the surgical site.

Additional objects, features and advantages of the various aspects ofthe present invention will become apparent from the followingdescription of its preferred embodiment, which description should betaken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a top view of an operating room employing a minimallyinvasive robotic telesurgical system.

FIG. 2 illustrates a block diagram of a telesurgical system.

FIGS. 3-6 illustrate block diagrams of telesurgical systems usingdifferent joint torque values for tool force indication.

FIG. 7 illustrates a block diagram of an observer useful in thetelesurgical system of FIG. 6.

FIG. 8 illustrates a flow diagram of a method for providing forceinformation to a user of a telesurgical system.

FIG. 9 illustrates a flow diagram of a method for providing forceinformation to a user of a telesurgical system with escalating warnings.

FIG. 10 illustrates a block diagram including components for providingauxiliary information related to a surgical procedure to be performed bya telesurgical system so as to be selectively displayable on a viewertogether with an image of a surgical site.

FIG. 11 shows a schematic view of an image of a surgical site displayedon the image display of the telesurgical system and further shows animage corresponding to auxiliary information from a selected source ofauxiliary information displayed in a window overlaid on the image of thesurgical site.

FIGS. 12A and B show schematic views illustrating the adjustment inposition and orientation of an image corresponding to auxiliaryinformation from a selected source of auxiliary information, relative toan image of the surgical site from an image capturing device.

FIG. 13 shows a schematic diagram of an image displayed at a viewer, andfurther shows a probe gathering auxiliary information relating to asurgical procedure.

DETAILED DESCRIPTION

FIG. 1 illustrates, as an example of a telesurgical system, a MinimallyInvasive Robotic Surgical (MIRS) system 100 including a Console (“C”)utilized by a Surgeon (“S”) while performing a minimally invasivediagnostic or surgical procedure, usually with assistance from one ormore Assistants (“A”), on a Patient (“P”) who is lying down on anOperating table (“O”).

The Console includes a support 102, a monitor 104 for displaying animage of a surgical site to the Surgeon, and one or more control devices108 (also referred to herein cumulatively as a “master manipulator”).The control devices 108 may include any one or more of a variety ofinput devices such as joysticks, gloves, trigger-guns, hand-operatedcontrollers, or the like.

The Surgeon performs a procedure by manipulating the control devices 108which in turn, cause robotic mechanisms 114 (also referred to herein as“slave manipulators”) to manipulate their respective removably coupledinstrument or tool assembly 110 (hereinafter simply referred to as a“tool”) through a minimally invasive incision in the body of the Patientwhile the Surgeon views the surgical site through the monitor 104.

To manipulate the tools 110, each of the slave manipulators 114 isconventionally formed of linkages that are coupled together andmanipulated through motor controlled joints. Since the construction andoperation of such robotic manipulators are well known, their detailsneed not be repeated here. For example, general details on roboticmanipulators of this type can be found in John J. Craig, Introduction toRobotics Mechanics and Control, 2^(nd) edition, Addison-WesleyPublishing Company, Inc., 1989.

The number of surgical tools 110 used at one time and consequently, thenumber of robotic mechanisms 114 in the system 100 will generally dependon the diagnostic or surgical procedure and the space constraints withinthe operating room among other factors. If it is necessary to change oneor more of the tools 110 being used during a procedure, the Assistantmay remove the tool 110 no longer being used at the time from itsrobotic mechanism 114, and replace it with another tool 110 from a tray(“T”) in the operating room.

The Surgeon's Console is usually located in the same room as the Patientso that the Surgeon may directly monitor the procedure, is physicallyavailable if necessary, and is able to speak to the Assistant(s)directly rather than over the telephone or other communication medium.However, it will be understood that the Surgeon can also be located in adifferent room, a completely different building, or other remotelocation from the Patient allowing for remote surgical procedures.

Preferably, control devices 108 will be provided with the same degreesof freedom as their associated tools 110 to provide the Surgeon withtelepresence, or the perception that the control devices 108 areintegral with the tools 110 so that the Surgeon has a strong sense ofdirectly controlling the tools 110. To this end, position, force, andtactile feedback sensors are preferably employed on the tools 110 totransmit position, force, and tactile sensations from the tools 110 backto the Surgeon's hands as he/she operates the control devices 108.

A monitor 104 is suitably coupled to a viewing scope assembly 112,including one or more cameras, through a processor 101, and positionedon the support 102 of the Console such that an image of the surgicalsite is provided near the Surgeon's hands. Preferably, the monitor 104will display a projected image on a display 106 that is oriented so thatthe surgeon feels that he or she is actually looking directly down ontothe operating site. To that end, an image of the tools 110 appear to belocated substantially where the operator's hands are located even thoughthe observation points (i.e., the endoscope or viewing camera) may notbe from the point of view of the image.

In addition, the real-time image is preferably projected into aperspective image such that the operator can manipulate the end effectorof a tool 110 through its corresponding control device 108 as if viewingthe workspace in substantially true presence. By true presence, it ismeant that the presentation of an image is a true perspective imagesimulating the viewpoint of an operator that is physically manipulatingthe tools 110. Thus, the processor 101 (or another processor in theConsole) transforms the coordinates of the tools 110 to a perceivedposition so that the perspective image is the image that one would seeif the viewing scope assembly 112 was located directly behind the tools110.

The processor 101 performs various functions in the system 100. Oneimportant function that it performs is to translate and transfer themechanical motion of control devices 108 to robotic mechanisms 114through control signals such as CS1 and CS2 so that the Surgeon (“S”)can effectively manipulate the tools 110. Another important function isto provide force information to one or more force indicators so that theSurgeon and/or Assistant(s) may be informed, for example, if excessiveforce is being applied by a monitored tool that may harm or causediscomfort to the Patient. In providing such force information, it isimportant that it is done in such a manner so as to not significantlyaffect the stability of the telesurgical system 100. In particular, itshould not drive the telesurgical system 100 unstable.

The force indicators, for example, may be integrated or attached to thesupport 102, and/or displayed on the monitor 104. Force indicators mayalso be activated on the control devices 108 in the form of vibration orviscous feel as described herein, provided the control devices 108 areequipped for such tactile sensations. Force indicators may also beplaced so as to be proximate to or positioned on their respective slavemanipulators 114.

The force information, for example, may be derived from strain gaugemeasurements on linkages in the slave manipulator manipulating the toolthat is being monitored, or it may be derived from encoders associatedwith joints in the slave manipulator manipulating the tool that is beingmonitored. Typical processing to generate the force information mayinclude filtering and/or gain adjustments.

The processor 101 may be separate from or integrated as appropriate intothe robotic mechanisms 114 and 115, it may be or be part of astand-alone unit, or it may be integrated in whole or in part into theConsole serving as its processor or as a co-processor to its processor.Although described as a processor, it is to be appreciated that theprocessor 101 may be implemented in practice by any combination ofhardware, software and firmware. Also, its functions as described hereinmay be performed by one unit, or divided up among different components,each of which may be implemented in turn by any combination of hardware,software and firmware.

FIG. 2 illustrates, as an example, a block diagram of a telesurgicalsystem 200 used in manipulating one of the tools 110 through itsrespective slave manipulator 114 in the MIRS system 100. The user 201 inthis case is the Surgeon (“S”) since it is the Surgeon (“S”) whomanipulates the master manipulator 108 in the MRS system 100.

As the user 201 manipulates the master manipulator 108, the slavecontroller 203 translates its position from the coordinate frame of themaster manipulator 108 to the coordinate frame of the tool 110. Theslave controller 203 then determines the joint positions for the slavemanipulator 114 that correspond to that tool position, and commandsmotors corresponding to each of those joints to move their respectivejoints to those positions using a closed-loop control system for each ofthe motors. Meanwhile, a master controller 207 feeds back any positionerror to the master manipulator 108 so that the master manipulator 108tends to move in tandem along with the slave manipulator 114.

The functions of the slave controller 203 and the master controller 207are implemented, for example, by programming them into a processor suchas the processor 101 in the MIRS system 100. An example showingadditional detail for such an implementation will now be described inreference to blocks 301-310 of FIG. 3. Referring to that figure, aclosed-loop control system for driving a joint motor in the slavemanipulator 114 is shown.

In this example, the closed-loop control includes a proportional,integral, derivative (“PID”) function 305 and a feed-forward (“FFD”)gain 304. Although a PID function is described herein, it is to beappreciated, however, that different control laws may also beimplemented and are fully contemplated to be within the full scope ofthe various aspects of the present invention. As indicated by the setsof arrows 302 and 309, the master manipulator 108 is understood to alsobe driving other similarly configured closed-loop control systemscorresponding to other joints of the slave manipulator 114.

The PID function 305 generates a feedback torque command (“TFBK”) byoperating on the joint position error between a commanded joint positionfrom the inverse Jacobian 301 (ignoring coordinate transformations) andthe detected joint position “Qx” from the joint encoder. The FFD gain304 generates a feed-forward torque command (“TFFD”) by operating on thecommanded joint position, velocity, and acceleration. The feedbacktorque (“TFBK”) and the feed-forward torque (“TFFD”) are then addedtogether to generate a total torque command (“TJ”) that is applied tothe joint motor, whose dynamics are depicted along with those of itsjoint in block 307, which is labeled JOINT DYNAMICS.

The joint position error is also provided to the master manipulator 108through a gain (“K”) 308 and transpose Jacobian 310. Although not shownto simplify the example, it is to be appreciated that a coordinatetransformation from slave joint space to Cartesian space is alsogenerally performed at this point. Since forces applied to the tool 110such as a static force experienced when the tool 110 is pressing againstan obstruction can create a joint position error, such reflected forcesare effectively passed back to the master manipulator 108 by suchposition error being fed back.

One problem with the part of the telesurgical system described so farwith respect to FIG. 2 is that additional filtering and/or gain toincrease the sensitivity for detecting certain forces on the tool isdifficult, since those changes may drive the joint closed-loop controlsystems incorporated therein to unstable conditions. As an example, if arelatively low level force is applied for an extended period of time bythe tool against an obstruction such as the Patient's rib-cage, it maynot be detected through the reflected forces being provided through theposition error that is fed back to the master manipulator 108 due to alow value of the gain “K” 308 that is required to maintain systemstability. As a consequence, bruising and/or prolonged discomfiture bythe Patient during and/or after the minimally invasive surgicalprocedure may result.

Accordingly, referring back to FIG. 2 now, a force indicator 209 andprocessing unit 208 are added to the telesurgical system 200 to providesuch types of tool force information to the user 201 without affectingthe stability of the closed-loop control systems in the telesurgicalsystem 200. In this case, the processing function 208 processes force ortorque information received from the slave controller 203 substantiallywithout restriction as to gain or filtering, because it is outside ofthe closed-loop control systems previously described herein.

As shown in FIGS. 3-6, the force or torque information from the slavecontroller 203 may be picked-off from several different points in thejoint motor control systems. For example, in FIG. 3, the total jointtorque (“TJ”) command provided to the joint motor may be picked-off forgenerating the force information to be provided to the user 201 throughthe force indicator 209. In FIG. 4, the feedback torque (“TFBK”)generated by the PID function 305 is picked-off for generating the forceinformation. In FIG. 5, the integrator torque (“TL”) from the integratorin the PID function 305 is picked-off for generating the forceinformation. In FIG. 6, an observed disturbance torque “TO” that isgenerated by an observer 601 is used for generating the forceinformation to be provided to the user 201 through the force indicator209. An example of the observer 601 is illustrated FIG. 7. Sinceobservers of this type are well-known in robotic control theory,detailed discussion of this figure is deemed unnecessary.

Note that depending upon the force that is to be presented to the user201, the picked-off force locations may differ for different joints ofthe slave manipulator 114, and only selected ones of the joints may betapped for picking off force or torque information. In addition, thegains and filters used for processing the picked-off force or torquevalues may be different for each of the joints. The processed forceinformation thus picked off the joint control systems for the selectedjoints are then combined in an appropriate fashion before providing theforce information to the user 201 through the force indicator 209.

The force indicator 209 may take any one of many different forms ormodalities that is preferably turned-on or activated and turned-off ordeactivated according to force threshold criteria. In the followingexamples, the force information is generated so as to determine a staticforce produced as the tool is pressed against an obstruction.

In one example of the force indicator 209, the force information may beprovided to the user by turning on a user-visible indicator wheninformation of the static force is greater than a first threshold value,and turning off the user-visible indicator when the information of thestatic force is less than a second threshold value. In this case, thefirst threshold value would generally be greater than the secondthreshold value.

One example of the user-visible indicator is a bar graph which may bedisplayed on the screen 106 of the monitor 104 of the MIRS system 100 sothat it is visible to the user of the telesurgical system. In this case,as the static force asserted against the tool increases, the length ofthe bar graph increases accordingly.

Another example of the user-visible indicator is a blinking icon on thescreen 106 of the monitor 104. Similarly, the user-visible indicator maybe a flashing light on the support 102 of the Console or on the mastermanipulator 108 of the MRS system 100 where the Surgeon would be able toreadily see it, or the flashing light may be on or in the proximity ofthe slave manipulator 114 of the MRS system 100 where the Surgeon and/orthe Assistant(s) may be able to see it.

The color of the user-visible indicator may also change as the staticforce increases, such as going from green (indicating a safe level offorce), to yellow (indicating a warning that the force is getting closeto an unsafe or undesirable level), and to red (indicating an unsafe orundesirable level of force has been reached). In addition oralternatively to a change in color, the intensity of the user-visibleindicator may change as the static force changes.

Another type of force indicator 209 is a user-audible indicator whichpreferably increases in intensity as the magnitude of the applied forceincreases. Another type of force indicator 209 uses haptic or tactilesensation features that may be implemented on the master manipulator108, such as a haptic “buzz” that provides a buzzing sensation to theSurgeon while manipulating the master manipulator 108 or a haptic“viscosity” that makes operation of the master manipulator 108 feel moresluggish to the Surgeon. In the case of these tactile sensations beingactivated on the master manipulator 108, the frequency and/or amplitudeof the “buzz” or the “viscosity” should be limited so as not tosubstantially affect the stability of the closed-loop control systems ofthe telesurgical system.

FIG. 8 illustrates a method for providing force information to the user201 which is implemented, for example, by the addition of the forceindicator 209 and the processing unit 208 to the telesurgical system200. In 801, torque values are determined for joints employed in thetelesurgical system for manipulating a tool. The torque values in thiscase are determined, for example, by the slave controller 203 processingthe movement of the master manipulator 108 as manipulated by the user201 (to determine TFFD, for example) and the movement of the joints ofthe slave manipulator 114 (to determine TFBK, for example).

The operation of the closed-loop controls systems and the providing offorce information to the user may then take place concurrently. Inparticular, in 802, the determined joint torque values are used in theirrespective closed-loop control systems, for example, as described inreference to blocks 301-310 of FIG. 3, while in 803, at least one of thetorque values is processed to generate force information for the tool,and in 804, the force information is provided to the user of thetelesurgical system in a manner so as not to significantly affect thestability of the joint closed-loop control systems.

Although the processing function 208 of the telesurgical system 200 isshown as being a simple gain and/or filter in corresponding blocks ofFIGS. 3-6, it is to be appreciated that the processing may take onadditional sophistication such as illustrated in FIG. 9. In particular,as shown in that figure, various force indications may be activated asthe static force asserted on the tool increases. At each level, theforce indication may be a different color or intensity as describedpreviously herein, or it may be a different modality. For example, thelevel 1 force indication may be a user-visible indication, the level 2force indication may be a user-audible indication, and the level 3 forceindication may be a tactile sensation on the master manipulator 108. Asin the cases of the slave controller 203 and the master controller 207,the processing function 208 is also implemented in a processor such asthe processor 101 in the MIRS system 100.

A method and system whereby auxiliary information related to a surgicalprocedure to be performed by the system 100 can be selectively displayedon the viewer 106, together with an image of the surgical site capturedby the endoscope 112, so as to enable the surgeon selectively toreference such information on the viewer 106 during the performance ofthe surgical procedure, in accordance with the invention, will now bedescribed.

By displaying auxiliary information related to the surgical procedure inthe image of the surgical site displayed at the viewer 106, the surgeonis able to reference such information without having to look at anothersource or display. For example, by displaying a patient's ECG signal inthe image together with the image of the surgical site captured by theendoscope 112, the surgeon need not transfer his direction of view to alocation removed from the image of the surgical site. This enables thesurgeon to perform the surgical procedure with greater ease andconfidence and with less distraction. Furthermore, the surgeon canprepare preoperative information specific to the surgical procedure tobe performed, or specific to the patient on which the surgical procedureis to be performed, so as to enable the surgeon selectively to accesssuch specific auxiliary information in the displayed image during theperformance of the actual surgical procedure. When displaying theauxiliary information together with the image of the surgical sitecaptured by the endoscope is referred to in this specification, such adescription is to be interpreted to have a wide meaning including, forexample, displaying the image in a discrete window overlaid on the imageof the surgical site, displaying the auxiliary information so as to bemerged with the image of the surgical site, such as merging apreoperative x-ray image with the image of the surgical site so that thesurgeon can view hidden detail of the surgical site, displaying theauxiliary information selectively on the viewer instead of the image ofthe surgical site so that the surgeon is presented with an unobstructedview of the surgical site when performing the surgical procedure, theauxiliary information then being selectively displayable in the image atthe viewer alternately with the image of the surgical site, and thelike. It will be appreciated that the auxiliary information can bedisplayed on a separate image display or viewer where appropriate.

Referring to FIG. 10 of the drawings, a plurality of sources oftwo-dimensional information is generally indicated by reference numeral312. Another plurality of sources of two-dimensional information isgenerally indicated by reference numeral 314.

The sources of two dimensional auxiliary information at 312 defineauxiliary information to be displayed in the image at the viewer 106 andwhich is of a type which, when displayed in the image, is to beadjustable to vary its displayed position relative to the image of thesurgical site captured by the endoscope. The imaged information from 312is typically adjustable relative to the image of the surgical site intwo dimensions only. Accordingly, the position of the imaged informationcan be varied to change its position across the image of the surgicalsite.

If the imaged information from 312 is displayed in a window overlaid onthe image of the surgical site, the size of the window is typically alsoadjustable in two dimensions. The types of information selectivelyaccessible from the sources 312 include, for example, a prerecordedstreaming video of the surgical procedure to be performed so that theoperator can follow the procedure as depicted in the video whiledisplayed in the image at the viewer 106 together with the image of thesurgical site. The types of information can further include, forexample, a real time ECG signal so that the surgeon can monitor thepatient's heart beat within the displayed image at the viewer 106.

Another type of auxiliary information can be in the form of a previouslycaptured and stored image from the endoscope of the surgical site,wherein the pre-captured image was taken to provide a generallypanoramic view of the surgical site and the surrounding scene. Such apre-captured panoramic image can be obtained by the endoscope 112. Insuch a case, the image can be captured when the viewing end of theendoscope 112 is relatively far removed from the surgical site. Afterthe panoramic image or view is captured in this fashion, the endoscopecan be moved such that its viewing end is closer to the surgical site soas to obtain a more suitable real time image for use in the performanceof the actual surgical procedure.

It will be appreciated that images other than a panoramic image of thesurgical site and surrounding scene can be provided for selectivereference on the image display at the viewer 106. Such other images caninclude, for example, generic or patient specific anatomical images foraiding the operator, or surgeon, for example, in identifying structuresso as to determine the surgical site location relative to the patientanatomy. Furthermore, such images can include, for example, imagesshowing the location of the entry ports, or incision points, theposition of the surgical instrument shafts and/or the end effectors soas to provide the operator with visible information relating to thelocation of surgical instruments or parts thereof. Such image can becomputer generated where appropriate, or can be obtained from additionalimage capture devices, and/or the like. This can be useful to avoidcollisions between the instrument shafts, for example. Furthermore, thiscan provide the operator with visible information enabling him toperceive how the instruments are interacting with each other and/or thepatient, in addition to the real time image of the surgical site used toperform the actual surgical procedure. When this information isselected, the auxiliary information can be displayed, where appropriate,to surround or abut a generally closer view of the surgical sitecaptured continually, or in real time, by the endoscope and which isused by the surgeon to monitor and control the surgical procedure. Inthis manner the surgeon, or operator, can be provided with the real timeimage from the endoscope at a preferably generally centrally disposedlocation in the viewed image, while the pre-captured, or real time,auxiliary image, e.g., a more panoramic view of the surgical site andsurrounding scene, is displayed along the periphery of the real timeimage obtained from the endoscope 112. This can serve to provide theoperator with a better idea of where he or she is operating relative tothe area surrounding the surgical site. Instead of providing theauxiliary image to surround the real-time image of the surgical site,the auxiliary image can be displayed in a discrete window, or in a“picture in picture” arrangement, extending over the image of thereal-time surgical site image. As another alternative, the auxiliaryimage can be displayed alternately with the actual real-time image.Thus, during the performance of a surgical procedure the surgeon canintermittently switch between the image of the real-time surgical siteimage and the auxiliary image by means of any appropriate switchinginput device or method, such as, buttons, switches, voice command,and/or the like. When the information from 312 is displayed in a windowoverlaid on the image of the surgical site, the surgeon can typicallyvary the size of the window and place the window relative to the imageof the surgical site so that the information is presented at a locationwhich is comfortable to the surgeon and at which the window does notobstruct important detail of the surgical site image.

By way of example, a specific application of such a “picture in picture”arrangement will now be described. During the course of a surgicalprocedure, the displayed image of the surgical site is typically in theform of a “narrow” field of view image normally being live, e.g.,continually updated, magnified and focused particularly on the surgicalsite. Such a “narrow” field of view typically provides the operator witha large image of a relatively small area in the patient. Such a “narrow”field image is typically captured in real time by means of the endoscope112. It has been found advantageous to provide the operator with a “wideangle” image of the surgical site and surrounding scene, to assist theoperator in determining where the surgical site and surgical tools arewith reference to the surrounding scene. Such a “wide angle” image canbe in the form of a “still” image captured by the same endoscope at aposition further removed from the surgical site than at which it isnormally positioned when capturing the real time image used by theoperator as he or she performs the surgical procedure. Instead, the“wide angle” image can be captured in real time by another image capturedevice, or endoscope, or the like. The two images can be displayed in avariety of different ways. In one way, the “wide angle” image can bedisplayed in a “smaller” window and the “narrow” field image can bedisplayed over a relatively larger area. The surgeon can then refer tothe “smaller” window for referencing orientation, or the like. Inanother way, the “narrow” field image is displayed in a “smaller” windowand the “wide angle” image is displayed over a relatively “larger” areato provide context to the surgeon to help him or her to remain orientedat the surgical site.

It can happen that the surgeon wishes to change the image displayed onthe viewer 106. This can be achieved, e.g., by rotation of the endoscope112 relative to the site viewed. Where the “wide angle” image is a“still” image, this image can be caused to rotate together with rotationof the “live”, magnified image. This can be achieved by causing the“still” image to be modified, for example, by means of computer control,so that the “still” image rotates to the same degree as the “live”image, so as to maintain, for example, context for the surgeon shouldthe surgeon desire to rotate the endoscope during surgery. In addition,or instead, if the surgeon desires to pan with the endoscope, the“still” image can be modified so that the “still” image preservesalignment, or registration, with a corresponding part of the “live”image.

The sources of two dimensional auxiliary information at 314 defineauxiliary information to be displayed in the image at the viewer 106 andwhich is of a type which, when displayed in the image, is to beadjustable to vary not only its two-dimensional displayed positionrelative to the image of the surgical site captured by the endoscope,but also its displayed orientation in three dimensions relative to thedisplayed image of the surgical site. One of the sources at 314 cancontain preoperative information which is to be aligned or brought intoregister with the image of the surgical site. For example, a twodimensional CAT scan image of a surgical site particular to the patienton which the surgical procedure is to be performed can be obtainedpreoperatively and loaded into one of the sources at 314. Such apreoperative image can be obtained so as to correspond with an image tobe captured by the endoscope, in other words, an image corresponding tothe image which the endoscope is to capture during the surgicalprocedure from a specific vantage point. Instead, the preoperative imagecan be from a vantage point different to that of where the endoscope isto be during the surgical procedure. During the surgical procedure, thesurgeon can then access the CAT scan information from the particularsource at 314 and place it in the displayed image of the surgical site.Such an image can then be adjusted in three dimensions so as to bringthe preoperative CAT scan image generally into register with the imageof the actual surgical site captured by the endoscope. Since theinformation from the sources 314 represent two dimensional information,there may be a limit to the amount of orientation change that can betolerated before the information ceases to be of use to the surgeon.

Still referring to FIG. 10 of the drawings, a plurality of sources ofthree-dimensional information is indicated at 316. One of the sourcescan include, for example, a three-dimensional model corresponding to asurgical site on which a surgical procedure is to be performed. Such athree-dimensional model can be, for example, raw volumetric images, suchas point cloud or voxcel representations, or the like, a computergenerated three-dimensional model or image, a segmentedthree-dimensional model obtained from CAT (Computer Aided Tomography)scans, MRI (Magnetic Resonance Imaging) techniques, or the like. Duringthe surgical procedure, the surgeon can then access the model and placeit in the image of the surgical site. The image corresponding to theauxiliary information in the form of the three-dimensional model, cantypically be superimposed, or merged, with the image of the surgicalsite. The brightness of the image of the three-dimensional model istypically adjustable so as to cause it selectively to fade relative tothe actual image of the surgical site.

Once placed in the image, the image of the model can be positionally andorientationally adjusted, and typically scaled, so as to enable thesurgeon to bring the preoperative image into register with the actualimage of the surgical site. Should the position of the endoscope bechanged, for example, to obtain an image of the surgical site from adifferent vantage point, the registration of the preoperative image canbe made to remain in register with the surgical site. This can typicallybe accomplished by causing the control system of the surgical system 100to fix the position of the preoperative image relative to a suitablereference frame once the surgeon has brought the preoperative imagegenerally into register in the displayed image. A suitable referenceframe can be, for example, a reference frame attached relative to thepatient, or the like. Since registration is often effected visually bythe surgeon, it may be that the registration is not entirely true oraccurate. Thus, should the endoscope position be moved to capture animage of the surgical site from a different vantage point, it may bethat the surgeon may again have to perform a slight adjustment to theregistration should the preoperative image not be correctly registeredwith the actual image of the surgical site upon changing the endoscopeposition. Instead of manual registration as described above, automaticregistration of the preoperative image with the surgical site image canbe achieved in accordance with known imaging techniques. Advantageously,registration can be accomplished by enabling the surgeon, or operator,to perform an initial manual registration procedure, followed by anautomatic registration procedure in accordance with conventional methodsto achieve a truer registration. Although reference has been made to amodel, it will be appreciated that other auxiliary information can beused instead. Such other auxiliary information can include preoperativeimages as well as inter-operative images. For example, aninter-operative image, or preoperatively obtained model, and/or thelike, of a beating heart can be registered with the actual image of thebeating heart as captured by the endoscope, and/or the like.

Referring again to the two-dimensional information at the sources 312,the two dimensional information can typically be in the form ofintrinsically two-dimensional information. Such information can includetwo dimensional visual images, such as video images, x-ray images,ultrasonic images, and/or the like. These two-dimensional images can bein digital or analog format, or the like. The information can be in theform of static images. Such static images can be in tiff, jpeg, and/orthe like, file formats, for example. The information can be in the formof moving images, such as, for example, streaming videos, as alreadymentioned. Such moving images can be in mpeg, digital video, analogvideo, such as NTSC or PAL, and/or the like, formats, for example. Theinformation can be textual, numeric, symbolic, and/or graphic in form.For example, the information sources can include sources of informationin the form of words, numeric readouts, status icons, bargraphs,stripchart displays, and/or the like. In this manner, for example,representations of blood pressure gauges, heartbeat rate, warmingmessages, notifications, warming lights, warning icons, or other warmingsignals related to system status, for example, the time in the form of arepresentation of a digital or analog clock, e-mail messages, and/or thelike, can be displayed. Accordingly, numeric readouts can correspond toblood pressure, heartbeat rate, elapsed and absolute time, and/or thelike. Status icons can include icons indicating the status of the system10, the identification of the type of surgical instruments currentlymounted on the robotic arms, and/or the like. Bar graphs can correspondto patient specific information, such as, temperature, oxygen levels inthe patient's blood, and/or the like. Bar graphs can also correspond tosystem specific information such as force magnitude, back-up batterystatus, and/or the like. Strip charts can correspond to EEG, ECG, bloodpressure, and/or the like. Symbolic or graphic representations cancorrespond to clocks, warning indicators, and icons selectivelyactivatable to provide access to sources of other auxiliary information,such as the three-dimensional and two-dimensional information, describedabove, menus, web pages and/or the like.

One, or more, of the sources may even comprise a separate computeroperatively connected to the system 100. The computer can be a computeron which a surgeon has prepared preoperative information for a specificpatient on which a surgical procedure using the system 100 is to beperformed. Such a computer may be remote from the system 100. Whenlinked to the system 100 as a source of auxiliary information, inaccordance with the invention, the surgeon is able to access suchpreoperative information on the remote computer from the system 100, soas selectively to display such information on the viewer 106 during theperformance of the surgical procedure. Thus, the surgeon, from thissource, can access information which may be resident on a computerscreen within his or her office, for example.

The images derived from the sources at 312, 314 and/or 316, may bestored images or may be real-time images. Accordingly, the system 100may include dedicated memory on which the images can be recordedpreoperatively if the images are patient or surgical site specific, forexample, so as to be stored in resident memory of the system 100.Instead, or in addition, the system 100 can have one or more inputconnectors, or jacks, to enable the system 100 to be operatively linkedto a source of auxiliary information external to the system 100. In thisfashion, the system can be linked to an external source of auxiliaryinformation, such as, for example, a remote computer as described above,an ECG source, computer networks such as Local Area Networks (LANS), theinternet, and/or the like. Accordingly, it will be appreciated that thesources 312, 314 and 316, can be in the form of resident memory of thesystem 100, on which memory the auxiliary information is stored, or canbe in the form sources external to the system 100, which externalsources are connectable to the system 100 through the input connectorsor jacks.

Sources of three-dimensional information are indicated at 316. Thesesources represent information which is intrinsically three-dimensional.Such types of information can include, for example, segmented organand/or vasculature models, patient specific and/or generic biomedicalmodels, non-biological geometric shapes, markers, and/or the like. Suchtypes of information can also include, for example, real timethree-dimensional video, laser scans, and/or the like. Such types ofinformation can yet further include landmarks, identifiers, or othermarkers that are attached to fixed locations in space. The use of suchlandmarks, identifiers, or other markers will now be described, by wayof example. In the case where the surgeon wishes to perform ananastomosis, for example, he or she can place a landmark, or identifier,or the like in the image displayed on the image display and then movethe landmark or marker to correspond with the area where the anastomosisis to be performed. The marker can then be attached to the area so thatif the endoscope is moved, for example, the marker remains in aregistered condition with the area to which it is attached.

The non-biological geometric shapes are typically used to place visiblehaptic constraints in the displayed image at the viewer 106. The purposeof placing such haptic constraints in the image is, for example, toinhibit the end effectors from moving beyond such constraints,containing end effector movement within such constraints, and/or thelike. Accordingly, the operator of the system can select anappropriately shaped geometric shape, or shapes, and, place it, or them,in the image, and then position the selected geometric shape, or shapes,in the image around an area, or organ, or tissue, for example, so as toprotect that area, or organ, or tissue from invasion by the endeffectors 110, or to constrain end effector movement to remain withinsuch shape or shapes, miter-box-fashion. Thus, should the site on whichit is desired to perform a surgical procedure be close to a sensitiveorgan, or tissue, or the like, an appropriately shaped geometric shape,or shapes, can be selected, placed in the scene of the surgical site andmoved into a position in which the selected shape, or shapes, extendover the sensitive area. When the shape, or shapes, is so placed, acorresponding haptic constraint, corresponding to the selected andplaced geometric shape, or shapes, is initialized so as to inhibit theend effectors 110 from trespassing beyond the visible constraint, orconstraints, as placed in the image by the surgeon thereby to protectthe sensitive tissue, or organ, or the like. The geometric shapes can beof any appropriate shape. Accordingly, such shapes can include, forexample, polyhedral shapes, NURBS (Non-Uniform Rational B-Spline),implicit surface shapes, planar shapes such as walls, and/or the like.The geometric shapes can include volumetric shapes such as point cloud,voxcels, and/or the like. The file formats used to store such geometricshapes can be .obj, .dxf, .3ds, VRML, and/or the like, for example. Itwill be appreciated that once an appropriate selected geometric shape,or shapes, is placed in the image, the surgeon can move the shape, orshapes, into a position covering or shrouding an area of sensitivity.When this has been done, the control system of the system 100 cantypically allocate coordinates to the placed shape, or shapes, relativeto an appropriate frame, such as a frame attached to the patient, or thelike. The system, after having determined the coordinates correspondingto the placed shape, or shapes, then inhibits the end effectors frommoving beyond such coordinates or constrains end effector movement toremain within such coordinates. For a more detailed description of acontrol system of the system 100 whereby such constraints can beimposed, refer to U.S. application Ser. No. 09/288,068 filed Apr. 7,1999 entitled “Aspects of a Control System of a Minimally InvasiveSurgical Apparatus”, now U.S. Pat. No. 6,493,608. Geometric shapes canalso be used to guide the surgeon or to assist in finding locations ofparticular interest. Furthermore, haptic feedback can be used toindicate information about objects which may not be readily discernablevisually. For example, sensitive areas can be given repulsive behaviorso that the tools are not only inhibited from approaching the sensitiveareas, but are restrained when approaching the sensitive areas at apredetermined distance from such areas.

Such geometric shapes can be provided with geometric description oradditional information, and can contain information about appearance,e.g., via visual texture mapping, and/or the like, surface and volumeproperties, e.g., such as mass, density, impedance, and/or the like, inaccordance with known methods in the field of haptics. The shapes canalso be derived from biological sources such as segmented MRIs. Suchadditional information about geometric shapes can be used for visualrepresentation, e.g., colors, patterns, textual maps, flashingappearances, and/or the like. Such additional information can also beused with haptic rendering to provide, for example, stiffness,artificial friction, masses, vibrations, or other physical ornon-physical force cues. The various sources of information as indicatedat 312, 314, and 316, are typically represented as icons on the displayarea of the video display 106. Accordingly, the operator of the systemcan select any one or more of the desired sources by selecting theappropriate associated icon. The step of selecting the desired source ofauxiliary information is indicated by the blocks 318, 320, and 322 forthe sources at 312, 314, and 316, respectively. Selection of a desiredsource typically takes place at the operator console C. Such selectioncan be made in any appropriate manner, such as by using buttons, footpedals, a mouse, and/or the like, for example. Advantageously, suchselection is made by making use of one, or both, or either of the mastercontrols 108, 108. In such a case, one, or both, or either, of themasters 108, 108 can serve as a two-dimensional or three-dimensionalmouse. Accordingly, one, or both, or either, of the masters can bearranged to perform functions relative to the displayed image in amanner analogous to a conventional mouse relative to a computer screen.Therefore, one, or both, or either, of the masters can be arranged toperform functions such as to point, highlight, move, select, and/or thelike.

The masters each typically have at least six degrees of freedom ofmovement. Accordingly, when used as a three-dimensional mouse, suchmaster can be arranged to control six variables, for example. Therefore,functions such as, shifting, rotating, panning, tilting scaling, and/orthe like, can be performed simultaneously when one, or both, or either,of the masters are used as a three-dimensional mouse, without anotherinput being required. In particular, for two-handed or two-masteroperation, any windows or overlays can be handled as “elastic” bodies,such that resizing, scaling, warping, and/or the like, can, for example,be controlled by pulling the masters apart, or the like. In this manner,the selected auxiliary information when displayed in the display imageof the viewer 106 can be positionally and orientationally adjusted inthree-dimensions in a three-dimensional environment, where appropriate,or where desired. The masters 108, 108 are typically provided with forcefeedback. The force feedback on the masters 108, 108 can be arranged toprovide functions related to auxiliary information selection, placement,orientational and positional movement, for example, to draw, or “suck”,the masters to an icon when an associated cursor is within apredetermined area around the icon, and/or the like. Refer to U.S.application Ser. No. 09/398,507, entitled “Master Having RedundantDegrees of Freedom,” filed Sep. 17, 1999, now U.S. Pat. No. 6,714,839,the full disclosure of which is incorporated herein by reference, forfurther information in connection with master control. Whatever methodand/or device used to make such selection, the selection step isindicated in the block 324 at 326 and as indicated by the dashed lines327. It will be appreciated that the block 324 represents selection andregulation steps that are performed by means of the appropriate inputs,such as the master control devices 108, 108, at the surgeon's console Cby the operator.

The steps whereby the information from the information sources 312 isselected and then presented or placed in the image at the video displaywill now be described in greater detail.

As mentioned, the selective placing of the auxiliary information fromthe sources 312 can be selectively caused to be displayed to extend atleast partially across an image display area of the viewer 106, such asin a localized window. When displayed on the display area, the positionat which the information is displayed relative to the display area canbe regulated or changed by the operator in two dimensions. Once adesired source is selected by the operator by operation of anappropriate input at 326, the desired source is selected at 318. Theinformation from that selected source is then forwarded to atwo-dimensional transform indicated at 328, as indicated by arrow 330.After the two-dimensional transform step at 328, the information is fedto a video mix and fade step at 332, as indicated by arrow 334. At theblock 332, the information from the selected source at 312 is mixed withthe video image captured by the endoscope 112. The video image capturedby the endoscope 112 is indicated by arrow 336. When the informationfrom the selected source at 312 is thus mixed with the image captured bythe endoscope 112, the combined images are forwarded to the videodisplay as indicated by arrow 338 so that both images are placed in theimage at the viewer 106.

Referring to FIG. 11 of the drawings, an image comprising a combinationor merger of the image from the endoscope and the selected source at 312is indicated generally by reference numeral 313. An image derived fromthe source at 312 is indicated at 318, and is shown as being overlaid onthe image from the endoscope indicated at 317. A row of icons isindicated by reference numerals 315. The source at 312 was selected byactuating a corresponding one of the icons 315.

Referring again to FIG. 10 and as indicated by the dashed line 140, thesurgeon or operator of the system 100 can regulate the two-dimensionaltransform at 328, as indicated at 342. This can be achieved in anyappropriate manner, such as through appropriate input devices such as,for example, buttons, toggles, joysticks, mice, and/or the like.Advantageously, one, or both, or either, of the master control devices108, 108 are used as the input device or devices whereby thetwo-dimensional transform 328 can be regulated. The representation ofthe combined images can be presented such that the information from theselected source 312 is cropped in a localized window, as indicated inFIG. 11 of the drawings, in the image displayed at the viewer 106.Accordingly, the image 317 captured by the endoscope 112 is positionedto extend across at least a major part of the display area, theinformation from the selected source at 312 being positioned in alocalized window overlaid on the image captured by the endoscope 112. Bymanipulation of the input at 342, the two-dimensional transform at 328is regulated to cause the window displaying the information from theselected source at 312, to be moved relative to the rest of the image,and to be placed where desired by the operator, as indicated by arrows Jand K in FIG. 11. Typically, the size of the window can be varied, aswell as its position relative to the rest of the image, as indicated byarrows L and M.

The video mix and fade step 332 is also regulatable by, for example, theoperator at the operator console C, or by another person, at a differentlocation, if appropriate. An appropriate input for performing suchregulation is indicated at 344 and is operatively connected as indicatedby the dashed lines 345 to the video mix and fade block at 332. Bymanipulation of the input at 344, the information from the source at 312can be faded relative to the image from the endoscope 112.Advantageously, the input at 344 is also performed by means of one, orboth, or either, of the master controls 108, 108.

Referring now to the information sources at 314, these sources providetwo dimensional information which, when displayed on the display area atthe viewer 106, can be regulated so as to change the position of suchinformation relative to the display area at the viewer in threedimensions, as described in greater detail herein below.

An appropriate one of the sources of two-dimensional information at 314can be selected in similar fashion to the selection of one of thesources at 312. Accordingly, the operator can select information from adesired source at 314 by manipulating the appropriate input at 326. Theselection step is indicated at 320. Once selected, the information fromthe desired source is forwarded to a two-dimensional tothree-dimensional transform indicated at 346. At the step 346, thetwo-dimensional information from the selected source at 314 is convertedto a three-dimensional representation. It is then passed through thethree-dimensional transform indicated at 348. The three-dimensionaltransform at 348 is regulatable by the operator as indicated at 350 andby the dashed line 352. This can typically be achieved by means of anyone or more of the inputs mentioned above. However, advantageously, theappropriate input is one, or both, or either, of the master controls108, 108. By means of the input at 350, typically the position,orientation and scale of the two-dimensional information from theselected source at 314, can be regulated to change its position,orientation and scale in three dimensions. It will be appreciated that,in this fashion, not only the position, but also the orientation of thetwo-dimensional image as displayed in the image as viewed at the viewer106 can be changed.

Once the operator has regulated the two-dimensional information by meansof the three-dimensional transform at 348, the information is passed toblock 354, where the information is transformed from a three-dimensionalrepresentation into a two dimensional representation. Thetwo-dimensional transform is indicated at 356. The two-dimensionaltransform is regulatable by the operator through the input 342 so as tochange the position of the information, as displayed in the image at theviewer 106, in two dimensions. It will be appreciated that thiscorresponds to changing the position of the image of the auxiliaryinformation from the source at 314 relative to the image of the surgicalsite. After regulation at 356, the information is passed to a video mixand fade block at 358, where it is mixed with the image from theendoscope 112 as indicated by arrow 336. As in the case with the videomix and fade block 332, the operator can cause the information to faderelative to the image captured by the endoscope 112 by means of theinput at 344. The image 336 from the endoscope 112 is combined with theinformation from the selected source at 314 and is then forwarded to theviewer 106 to be displayed thereon.

Referring to FIG. 12A of the drawings, an image comprising a combinationor merger of the image from the endoscope and the selected source at 314is indicated generally by reference numeral 321. An image derived fromthe source at 314 is indicated at 323 and is shown as being overlaid onthe image from the endo scope indicated at 327. As in the case withreference to FIG. 11, and as can best be seen in FIG. 12B of thedrawings, the image from the source 314 can be repositioned withreference to arrows J and K and can be adjusted in size as indicated byarrows L and M. This is achieved by the operator of the system 100 at342 by means of the transform at 356 as indicated by dashed line 140.

In addition, and with specific reference to FIG. 12B of the drawings,the image from the selected source at 314 is orientationally adjustableor regulatable. Accordingly, the image from the selected source 314 canbe regulated so as to change its orientation in three dimensions withreference to the arbitrary reference frame indicated in dashed lines inFIG. 12A. Although in FIG. 12B the image from the source 314 is shown ashaving been adjusted angularly about an arbitrary y axis with referenceto the reference frame in FIG. 12A, it will be appreciated that angularadjustment about the x and z axes can be performed in similar fashion.Such angular regulation of the image from the selected source at 314 isachieved by the operator of the system 100 at 350, so as to regulate theinformation from the selected source at 314 by means of the transform at348 as indicated by dashed line 352. In similar fashion, the image canalso be moved “inwardly” and “outwardly” as indicated by arrows Q alongthe z-axis.

Referring now to the three-dimensional information sources 316,information from one or more of the sources can be selected by theoperator by means of the input 326 and as indicated by the block 322.The three-dimensional information from the selected source at 316 isthen passed to a three-dimensional transform as indicated at 362. Theoperator, by using the input device at 350, can then regulate thisinformation in the three-dimensional transform at 362 so as to varytypically the orientation, position and scale of an image derived fromthe selected source and as displayed at the viewer 106 in similarfashion as described above with reference to FIGS. 12A and 12B. Once theinformation has been regulated in this fashion, the information isforwarded to a block 364 where the three-dimensional information istransformed from three dimensions to two dimensions. The resultanttwo-dimensional information is then forwarded to a two-dimensionaltransform at 366. The information can then again be regulated by theoperator by means of the input device at 342 as herein before describedwith reference to the two-dimensional transforms 328, 356. As before,the resultant information is then fed to a video mix and fade block asindicated at 368 where the information is mixed with the image from theendoscope and is then passed to the viewer. Where appropriate, theinformation can be caused automatically to register with a correspondingsurgical site image captured by the endoscope as already describedherein above. Instead, as described above, registration can be manualonly, or a combination of manual and automatic methods.

It will be appreciated that the above methods can be used withtwo-dimensional single channel video display or with three-dimensionaldual channel video display. In the latter case, the real time videosource 336 can comprise two separate images for “right” and “left”channels for viewing by the right and left eyes of the surgeon. Elements354 and 364 can then provide two separate images from two distinctviewpoints for the right and left channels respectively. The subsequentelements, or steps, can then be applied to both channels. Furthermore,element 328 can be arranged to duplicate the signal 334 into a left anda right channel and to shift them relative to each other to place theoriginal two-dimensional image in a three-dimensional viewer at variableapparent depths.

Advantageously, at least one of the master controls is operativelyarranged to fulfill some, preferably all, of the functions in the block324. Accordingly, the operator need then not remove his hands from themaster control devices 108, 108 when selecting and changing theposition, orientation and scale of the auxiliary information whendisplayed in the image at the viewer 106. In this way, continuity ofcontrol of the surgical procedure is enhanced whilst still enabling theoperator to access and place auxiliary information from one or more ofthe sources 312, 314 and 316.

As already mentioned, the masters 108, 108 are normally operativelyassociated with the slaves. Typically, when one, or both, or either, ofthe masters are to be used selectively to place an image correspondingto auxiliary information from a selected source 312, 314, 316 in theimage or scene of the surgical site, the operative association betweenthe master, or masters, and the slaves is temporarily interrupted. Whenthis occurs, the slaves are typically held or locked in stationarypositions at the surgical site. Accordingly, the slaves are locked inthe positions they occupied immediately before disassociation with themasters 108, 108. The master or masters are then freed to enable them tobe used to select and place the desired auxiliary information in thescene or image of the surgical site captured by the endoscope 112 anddisplayed across the display area of the image display or viewer 106.Once the auxiliary information has been selected and placed, operativeassociation between the masters 108, 108 and the slaves isre-established to permit the operator to proceed with the surgicalprocedure with reference to the auxiliary information now displayed onthe display area of the viewer 106 after having been selected and placedin the scene by means of one, or both, or either, of the masters 108,108. Refer to U.S. application Ser. No. 09/398,960, entitled“Repositioning and Orientation of Master/Slave Relationship in MinimallyInvasive Telesurgcry,” filed Sep. 17, 1999, now U.S. Pat. No. 6,459,926,the full disclosure of which is incorporated herein by reference, for amore detailed explanation of how the operative association between themasters and the slaves is preferably reestablished.

When one of the masters is used to select the desired auxiliaryinformation, a cursor is typically generated in the image upondisassociation with the slaves. The cursor is then guided by movement ofthe master until the cursor is over the desired icon 315. The master isthen also typically used to actuate the icon to cause the desiredauxiliary information to be accessed and placed in the image of thesurgical site. When placed, the master, or both masters, is then used tovary the position and/or orientation of the image corresponding to theselected auxiliary information relative to the image of the surgicalsite as described above, and where appropriate. One or both masters maybe used to vary the position and orientation of auxiliary information,overlays and windows in a manner similar to the way in which masters areused to vary the position and orientation of an image from an imagecapture device. Of course, the present invention also encompasses othermanners of manipulating auxiliary information, in addition to thepreferred masters disclosed, such as by repositioning/rotating ajoystick, using multiple input buttons to indicate the desiredmanipulation, or using a voice control/recognition system to command thesystem to manipulate the auxiliary information as desired.

Should, during the course of a surgical procedure, an image capturedevice generating a real time video image 336 be moved, the imagedisplayed on the image display may be caused to shift and/or rotate inresponse to such image capture device movement. Instead, the video image336 can be caused to shift/rotate electronically, for example. Duringsuch a change in the displayed real time image, the two-dimensional andthree-dimensional transforms 328, 348, 354, 356, 362, 364, 366 can bearranged to synchronize their operation with the change in the displayedimage so as to cause the auxiliary information to appear attached to thedisplayed real time image. Instead, the transforms can be arranged toignore the change in the displayed real time image to cause theauxiliary information to appear attached to the image display and todrift relative to the changing real time image.

Another source of auxiliary information will now be described withreference to FIG. 13. Such a source of auxiliary information cantypically include an appropriate image gathering device such as oneincluding a transmitter and receiver arrangement, as schematicallyindicated at 413. An example of such a device is an ultrasoundtransducer which will be used by way of example only in the descriptionwhich follows. Accordingly, the invention is not to be limited to anultrasonic device. Any appropriate device which can gather similarinformation falls within the scope of the invention. Such a source canbe used to obtain a preoperative or intraoperative two-dimensional orthree-dimensional image, or model, corresponding to a surgical procedureto be performed. Accordingly, it can be either a two-dimensional source312, 314 or a three-dimensional source 316 depending on its application.As a two-dimensional source, the ultrasonic transducer can be used toobtain a single ultrasound image. As a three-dimensional source it canbe used to obtain a plurality of spaced ultrasonic images, or cuts,thereby to provide sufficient information for construction of athree-dimensional model. Accordingly, it can be arranged to move, orsweep, across a surgical site to capture such images, or cuts. This cantypically be achieved, for example, in accordance with a pre-programmedsequence for moving the ultrasound transducer, manual movement of theultrasound transducer, or the like. The ultrasonic transducer can bemounted at an end of a shaft to enable it to be introduced to thesurgical site through a relatively small aperture, in a minimallyinvasive manner. The sweeping movement can be performed manually bymoving an opposed end of the shaft positioned outside the body. To thisend, a handle can be provided on the opposed end of the shaft.Conveniently, manually operable actuators can be provided at the handleto enable the ultrasonic transducer, or probe, to be moved relative tothe end of the shaft on which it is mounted by manipulating theactuators. Instead, the shaft can be mounted on a robotic arm, themovement being controlled through a master control device. In anotherembodiment, the movement of the ultrasonic transducer can be controlledby means of a computer program. Accordingly, whether performed manuallyor automatically, a plurality of separate images can be obtained andused to form a “mosaiced” surface of images in a fashion similar to thatknown in the satellite and undersea imaging industries, namely, by“painting” the sensor, or ultrasonic transducer, across the surfacebeing viewed. Said surface of images may be intrinsically two- orthree-dimensional in nature depending on the movement of the sensorduring the build-up of the image. A different series of image “slices”may be constructed from a sensor that produces a planar image and thatis moved substantially normal to the image plane to produce a series ofslices, as is known, for example, in prenatal ultrasonic imagingpractice. Taken together, these form an intrinsically three-dimensionalor volumetric image.

These built-up two- and three-dimensional images may then be introducedinto the system to be selectively overlaid and positioned within thesurgeon's field of view at the viewer. As can best be seen in FIG. 13,such an ultrasonic image, when in a two-dimensional format, may bedisplayed as indicated by reference numeral 411.

Such a source can also be used inter- or post-operatively. For example,it can be used as a flow probe, or the like, to enable the surgeon, forexample, to ascertain the degree of fluid flow through a vessel, or thelike. In such a case, when, for example, an anastomosis procedure hasbeen performed, a surgeon, or operator, of the system may wish todetermine whether or not the anastomosed vessels are allowing sufficientblood flow therethrough, whether or not one or more of the vessels hasbeen damaged during the procedure so as to require further correctivesurgery, and/or the like. The flow probe, or ultrasonic transducer, canthen be used to establish this.

Advantageously, the ultrasonic transducer, or other appropriate device,or flow probe, can be mounted on an end of a shaft 415 to permit it tobe introduced into a patient body in similar fashion to the surgicalinstruments 110, in a minimally invasive manner. The ultrasonictransducer 413 can be mounted on an end of the shaft 415 by means of thewrist member 417 to enable it to be angularly displaced relative to theshaft in multiple degrees of freedom of movement. The mounting of theultrasonic device on the end of the shaft, whether by means of one ormore wrist members, or otherwise, is preferably such as to provide theultrasonic device with relatively large sweeping movement capabilityrelative to the end of the shaft, as indicated by arrows 419.Accordingly, it can have a relatively large lateral range of motionalthough narrow ranges of motion, or none at all, relative to the end ofthe shaft, fall within the scope of the present invention. Movement ofthe ultrasonic device relative to at least the end of the shaft ispreferably controlled from outside the patient body, in use. For exampleactuators positioned remote from the end on which the ultrasonictransducer is mounted may be used to control movement of the ultrasonicdevice relative to the end of the shaft from outside the patient body.Instead, or in addition, actuators can be provided to cause theultrasonic transducer to scan an area of interest. The shaft may have ahandle at its proximal end, opposed from the flow probe, for manualcontrol by means of manually controllable actuators, or it may bemountable on a robotic arm as described above for control by means of amaster control device. Accordingly, in a preferred embodiment, theultrasonic device is mounted on a distal end of a robotic surgical toolof the type disclosed in U.S. Pat. No. 5,808,665, entitled “EndoscopeSurgical Instrument and Method For Use,” the full disclosure of which isincorporated herein by reference. Movement of the ultrasonic transduceracross a desired area of interest could then be accomplished by asurgeon or operator of the system 100 by manipulation of a remotelycontrolled master control at the control station C as described in U.S.application Ser. No. 09/398,507. Instead, the probe could be arranged tobe releasably grasped by a surgical instrument having an appropriatecomplimentary end effector.

Another application of the information gathered by such an ultrasoundprobe, or the like, is to collect preoperative data on the patient, atthe surgical site, for example. Such preoperative data can then be usedto determine a location of, for example, a stenosis, or blockage, or thelike, in a blood vessel that is to be anastomosed during a heart bypassoperation for example. The auxiliary information can then be overlaid onthe “live” image of the surgical site to indicate to the surgeon wherethe surgeon should conduct the anastomosis. Conveniently, and as alreadydescribed, markers or identifiers can then be attached to the locationof the stenosis such that, should the displayed image be changed, suchas, for example by moving the endoscope, the markers or identifiersremain in a registered condition with the stenosis so that the locationof the stenosis remains clearly indicated in the displayed image.

Although the various aspects of the present invention have beendescribed with respect to a preferred embodiment, it will be understoodthat the invention is entitled to full protection within the full scopeof the appended claims.

1-50. (canceled)
 51. A medical system comprising: a medical instrumenthaving a shaft and an end effector, the end effector coupled to a distalend of the shaft; an image capture device disposed relative to themedical instrument so as to capture real-time images of the end effectorand the distal end of the shaft, but not a proximal end of the shaft; aviewer displaying the real-time images; and a processor programmed tocause a computer generated image of the proximal end of the shaft to bedisplayed adjacent the displayed real-time images on the viewer so thatthe displayed computer generated image of the proximal end of the shaftappears to be connected to the real-time image of the distal end of theshaft.
 52. The medical system of claim 51, wherein the processor isprogrammed to display the computer generated image of the proximal endof the shaft on the viewer so that the position of the displayedcomputer generated image of the proximal end of the shaft relative tothe position and orientation of the distal end of the shaft seen in thedisplayed real-time images on the viewer corresponds to a position andorientation of the proximal end of the shaft that would be seen in apanoramic view of the medical instrument.
 53. The medical system ofclaim 51, wherein the processor is programmed to register the computergenerated image of the proximal end of the shaft to the real-time imagesof the distal end of the shaft so that the position and orientation ofthe displayed computer generated image of the proximal end of the shaftrelative to the position and orientation of the distal end of the shaftas seen in the displayed real-time images on the viewer indicates theposition and orientation of the proximal end of the shaft relative tothe distal end of the shaft.
 54. A medical system comprising: an imagecapture device capturing real-time images of a work site; a viewerdisplaying the real-time images; a medical instrument having a portionthat is not seen in the real-time images; and a processor programmed tocause a computer generated image of the portion of the medicalinstrument to be displayed adjacent the displayed real-time images onthe viewer so that a position and orientation of the computer generatedimage of the portion of the medical instrument on the viewer indicatesthe position and orientation of the portion of the medical instrumentrelative to the work site.
 55. The medical system of claim 54, whereinthe processor is programmed to display the computer generated image ofthe portion of the medical instrument on the viewer so that thedisplayed position and orientation of the computer generated image ofthe portion of the medical instrument corresponds to a position andorientation of the medical instrument that would be seen in a panoramicview of the work site that includes the portion of the medicalinstrument.
 56. The medical system of claim 54, wherein the processor isprogrammed to register the computer generated image of the portion ofthe medical instrument to the real-time images of the work site so thatthe position and orientation of the displayed computer generated imageof the portion of the medical instrument indicates the position andorientation of the medical instrument relative to the work site.
 57. Themedical system of claim 54, wherein the medical instrument has a shaftand an end effector, the end effector coupled to a distal end of theshaft, images of the end effector and the distal end of the shaftincluded in real-time images, the portion of the medical instrument notseen in the real-time images comprising a proximal end of the shaft. 58.A medical system comprising: an image capture device having a field ofview and capturing real-time images of a work site within the field ofview; a viewer displaying the real-time images; a medical device havinga first portion and a second portion, wherein the first portion isoutside the field of view of the image capture device, and wherein thesecond portion is within the field of view of the image capture device;and a processor programmed to cause a computer generated image of thefirst portion to be displayed adjacent the displayed real-time images onthe viewer so that the displayed computer generated image of the firstportion appears to be a continuation of the second portion seen in thedisplayed real-time images on the viewer.
 59. The medical system ofclaim 58, wherein the image capture device comprises an endoscope. 60.The medical system of claim 58, wherein the processor is programmed todetermine a position and orientation to display the computer generatedimage of the first portion on the display by determining the positionand orientation of the second portion relative to the captured real-timeimages of the work site and by using information of a construction ofthe medical device.
 61. The medical system of claim 58, wherein themedical device comprises a medical instrument having a shaft and an endeffector, the end effector coupled to a distal end of the shaft, and thesecond portion of the medical device comprising the end effector and thedistal end of the shaft.